The present invention relates to a method and apparatus for controlling quality of moving image reproduction in a three-dimensional virtual space.
Conventionally, control of reproduction quality such as a frame rate or image quality of a moving image in reproduction of the same, is performed mainly for data on a transmission line, and the moving image transmitted therethrough is the one reproduced according to the frame rate or the image quality set when transmitted.
By the way, in recent years, multimedia data such as the moving image, audio, and three-dimensional computer graphics have been provided through the use of Internet, CD-ROM, DVD, and the like, and contents which comprise the moving image and the three-dimensional computer graphics, fused with each other, and the moving image of which is reproduced in the three-dimensional virtual space, have been created.
When the moving image is reproduced at a spot distant from a view point of a viewer in this three-dimensional virtual space, its display size in a display projection plane is small. In general, when the display size is small, degradation of the moving image reproduction quality is hardly recognized. Hence, when the display size is small, it is not necessary to keep reproduction quality as high as that of the moving image of a large display size. That is, if the frame rate or the image quality is reduced according to the distance from the view point, the viewer can see the image on the display projection plane without any problem.
However, in a conventional moving image reproduction method, since reproduction of the moving image in the three-dimensional virtual space is not taken into account, it is required that the image be reproduced according to the set frame rate or image quality even when the display size is small as described above. The requirement that the same amount of operation be performed regardless of the display size is problematic.
The present invention is directed to solving the above problem, and an object of the present invention is to provide a method and apparatus for controlling reproduction quality of the moving image reproduced in the three-dimensional virtual space according to the distance from the view point.
According to one embodiment of the present invention, there is provided an apparatus for controlling the reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer in a three-dimensional virtual space. In this embodiment, the apparatus comprises: an object veneration means that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; object position determination means that receives data relating to the object placed in the three-dimensional virtual space as an input, moves the object according to an external user input and thereby determines the object position in the three-dimensional virtual space; view point position determination means that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; distance calculation means that receives the object position and the view point position as inputs and calculates a distance from the view point to the object, frame rate determination means that receives the calculated distance as an input, and determines a frame rate corresponding to the input distance on the basis of a preset table or formula showing a relationship between distances from the view point to the object and frame rates of the moving image to be mapped to the object; frame rate control means that receives the determined frame rate as an input and performs control so that a frame of the moving image to be mapped to the object is reproduced at the determined frame rate; moving image generation means that receives moving image data input externally, and reproduces a necessary frame of the moving image from the moving image data in accordance with the control; and perspective projection display means that receives the object position, the view point position, and the frame of the reproduced moving image as inputs, projects the object on a two-dimensional display projection plane based on the object position and the view point position, maps the frame of the moving image to a projection plane of the object, and displays the resulting image.
According to a second embodiment of the present invention, further to the apparatus for controlling moving image reproduction quality of the first embodiment, the distance calculation means further receives coordinates on the display projection plane in the three-dimensional virtual space as an input, and finds an inclination angle of the object with respect to the display projection plane based on the object position, the view point position, and the coordinates on the display projection plane in the three-dimensional virtual space, and the frame rate determination means updates the frame rate determined according to the distance into a frame rate corresponding to the distance and the inclination angle, on the basis of a preset table or formula showing a relationship between inclination angles and frame rates of the moving image to be mapped to the object.
According to a third embodiment of the present invention, there is provided an apparatus for controlling reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer graphics in a three-dimensional virtual space. The apparatus of this embodiment comprises: object generation means that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; object position determination means that receives data relating to the object placed in the three-dimensional virtual space as an input from the object generation means, moves the object according to an external user input, and thereby determine the object position in the three-dimensional virtual space; view point position determination means that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; perspective projection means that receives the object position and the view point position as inputs and projects the object on a two-dimensional display projection plane corresponding to the object position and the view point position; area calculation means that receives data relating to a projection plane of the perspectively projected object, and calculates an area of the projection plane; frame rate determination means that receives the calculated area as an input, and determines a frame rate corresponding to the input area with reference to a preset table or formula showing a relationship between areas of the projection plane of the object and frame rates of the moving image to be mapped to the projection plane; frame rate control means that receives the determined frame rate as an input and performs control so that a frame of the moving image to be mapped to the object is reproduced at the determined frame rate; moving image generation means that receives moving image data input externally, and reproduces a necessary frame of the moving image from the moving image data in accordance with the control; and display means that maps the frame of the moving image input from the moving image generation means to the two-dimensional display projection plane on which the object has been perspectively projected as an input from the perspective projection means, and displays the resulting image.
According to a fourth embodiment of the present invention, further to the apparatus for controlling moving image reproduction quality of the first embodiment or the third embodiment, the area calculation means further receives the object position as an input from the perspective projection means, and the frame rate determination means, when three-dimensional object data whose object size has been changed is input externally to the object generation means, further receives the object position as an input from the area calculation means, and recognizes that the object size has been changed, from the object position, and updates the frame rate determined according to the area into a frame rate corresponding to the changed object size, on the basis of a preset table or formula showing a relationship between object sizes and frame rates of the moving image to be mapped to the object.
According to a fifth embodiment of the present invention, there is provided an apparatus for controlling the reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer graphics in a three-dimensional virtual space. The apparatus of this embodiment comprises: object generation means that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; object position determination means that receives data relating to the object placed in the three-dimensional virtual space as an input from the object generation means, moves the object according to an external user input, and thereby determines the object position in the three-dimensional virtual space; view point position determination means that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; distance calculation means that receives the object position and the view point position as inputs and calculates a distance from the view point to the object; filter selecting means that receives the calculated distance as an input and selects a filter corresponding to the input distance on the basis of a preset table or formula showing a relationship between distances from the view point to the object and filters used for enlarging or reducing the moving image to be mapped to the object; filter control means that receives information about the filter selected by the filter selecting means as an input and performs control so that a frame of the moving image to be mapped to the object is enlarged or reduced by using the selected filter; moving image generation means that receives moving image data input externally and reproduces the frame of the moving image from the moving image data, filtering means that receives the frame of the moving image as an input from the moving image generation means and filters the frame in accordance with the control executed by the filter control means; and perspective projection display means that receives the object position, the view point position, and the filtered frame of the moving image as inputs, projects the object on a two-dimensional display projection plane corresponding to the object position and the view point position, maps the frame of the moving image to a projection plane of the object, and displays the resulting image.
According to a sixth embodiment of the present invention, further to the apparatus for controlling moving image reproduction quality of the fifth embodiment, the distance calculation means further receives coordinates on the display projection plane in the three-dimensional virtual space as an input, and finds an inclination angle of the object with respect to the display projection plane based on the object position, the view point position, and the coordinates on the display projection plane in the three-dimensional virtual space, and the filter selecting means updates the filter selected according to the distance into a filter corresponding to the distance and the inclination angle, on the basis of a preset table or formula showing a relationship between inclination angles and filters used for enlarging or reducing the moving image to be mapped to the object.
According to a seventh embodiment of the present invention, there is provided an apparatus for controlling the reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer graphics in a three-dimensional virtual space. The apparatus of this embodiment comprises: object generation means that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; object position determination means that receives data relating to the object placed in the three-dimensional virtual space as an input from the object generation means, moves the object according to an external user input, and thereby determines the object position in the three-dimensional virtual space; view point position determination means that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; perspective projection means that receives the object position and the view point position as inputs and projects the object on a two-dimensional display projection plane based on the object position and the view point position; area calculation means that receives data relating to a projection plane of the perspectively projected object, and calculates an area of the projection plane; filter selecting means that receives the calculated area as an input and selects a filter corresponding to the input area on the basis of a preset table or formula showing a relationship between areas of the projection plane of the object and filters used for enlarging or reducing the moving image to be mapped to the projection plane; filter control means that receives information about the selected filter and performs control so that a frame of the moving image to be mapped to the object is enlarged or reduced by using the selected filter; moving image generation means that receives moving image data input externally, and reproduces the frame of the moving image from the moving image data, filtering means that receives the frame of the moving image as an input from the moving image generation means and filters the frame in accordance with the control executed by the filter control means, and display means that maps the frame of the moving image input from the moving image generation means to the two-dimensional display projection plane on which the object has been perspectively projected as an input from the perspective projection means, and displays the resulting image.
According to an eighth embodiment of the present invention, further to the apparatus for controlling moving image reproduction quality of the fifth embodiment or the seventh embodiment, the area calculation means further receives the object position as an input from the perspective projection means, and the filter selecting means, when three-dimensional object data whose object size has been changed is input externally to the object generation means, further receives the object position as an input from the area calculation means, recognizes that the object size has been changed, from the object position, and updates the filter selected according to the area into a filter corresponding to the changed object size, on the basis of a preset table or formula showing a relationship between object sizes and filters used for enlarging or reducing the moving image to be mapped to the object.
According to a ninth embodiment of the present invention, there is provided a method for controlling the reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer graphics in a three-dimensional virtual space. The method of this embodiment comprises: an object generation step that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; an object position determination step that receives data relating to the object placed in the three-dimensional virtual space as an input, moves the object according to an external user input, and thereby determines the object position in the three-dimensional virtual space; a view point position determination step that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; a distance calculation step that receives the object position and the view point position as inputs and calculates a distance from the view point to the object; a frame rate determination step that receives the calculated distance as an input, and determines a frame rate corresponding to the input distance on the basis of to a preset table or formula showing a relationship between distances from the view point to the object and frame rates of the moving image to be mapped to the object; a frame rate control step that receives the determined frame rate as an input and performs control so that a frame of the moving image to be mapped to the object is reproduced at the determined frame rate; a moving image generation step that receives moving image data input externally, and reproduces a necessary frame of the moving image from the moving image data in accordance with the control; and a perspective projection display step that receives the object position, the view point position, and the reproduced frame of the moving image as inputs, projects the object on a two-dimensional display projection plane based on the object position and the view point position, maps the frame of the moving image to a projection display of the object, and displays the resulting image.
According to a tenth embodiment of the present invention, further to the method for controlling moving image reproduction quality of the ninth embodiment, the distance calculation step further receives coordinates on the display projection plane in the three-dimensional virtual space as an input, and finds an inclination angle of the object with respect to the display projection plane based on the object position, the view point position, and the coordinates on the display projection plane in the three-dimensional virtual space, and the frame rate determination step updates the frame rate determined according to the distance into a frame rate corresponding to the distance and the inclination angle, on the basis of a preset table or formula showing a relationship between inclination angles and frame rates of the moving image to be mapped to the object.
According to an eleventh embodiment of the present invention, there is provided a method for controlling the reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer graphics in a three-dimensional virtual space. The method of this embodiment comprises: an object generation step that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; an object position determination step that receives data relating to the object placed in the three-dimensional virtual space as an input from the object generation step, moves the object according to an external user input and thereby determines the object position in the three-dimensional virtual space; a view point position determination step that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; a perspective projection step that receives the object position and the view point position as inputs and projects the object on a two-dimensional display projection plane based on the object position and the view point position; an area calculation step that receives data relating to a projection plane of the perspectively projected object and calculates an area of the projection plane; a frame rate determination step that receives the calculated area as an input, and determines a frame rate corresponding to the input area on the basis of a preset table or formula showing a relationship between areas of the projection plane of the object and frame rates of the moving image to be mapped to the projection plane; a frame rate control step that receives the determined frame rate as an input and performs control so that a frame of the moving image to be mapped to the object is reproduced at the determined frame rate; a moving image generation step that receives moving image data input externally, and reproduces a necessary frame of the moving image from the moving image data in accordance with the control; and a display step that maps the frame of the moving image as an input from the moving image generation step to the two-dimensional display projection plane on which the object has been perspectively projected as an input from the perspective projection step, and displays the resulting image.
According to a twelfth embodiment of the present invention, further to the method for controlling moving image reproduction quality of the ninth embodiment or the eleventh embodiment, the area calculation step further receives the object position as an input from the perspective projection step, and the frame rate determination step, when three-dimensional object data whose object size has been changed is given to the object generation step as an external input, further receives the object position as an input from the area calculation step, recognizes that the object size has been changed, from the object position, and updates the frame rate determined according to the area into a frame rate corresponding to the changed object size, on the basis of a preset table or formula showing a relationship between object sizes and frame rates of the moving image to be mapped to the object.
According to a thirteenth embodiment of the present invention, there is provided a method for controlling the reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer graphics in a three-dimensional virtual space. The method of this embodiment comprises: an object generation step that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; an object position determination step that receives data relating to the object placed in the three-dimensional virtual space as an input from the object generation step, moves the object according to an external user, and thereby determines the object position in the three-dimensional virtual space; a view point position determination step that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; a distance calculation step that receives the object position and the view point position as inputs and calculates a distance from the view point to the object; a filter selecting step that receives the calculated distance as an input and selects a filter corresponding to the input distance on the basis of a preset table or formula showing a relationship between distances from the view point to the object and filters used for enlarging or reducing the moving image to be mapped to the object; a filter control step that receives information about the filter selected in the filter selecting step as an input and performs control so that a frame of the moving image to be mapped to the object is enlarged or reduced by using the selected filter; a moving image generation step that receives moving image data input externally and reproduces the frame of the moving image from the moving image data; a filtering step that receives the frame of the moving image as an input from the moving image generation step and filters the frame in accordance with the control executed in the filter control step; and a perspective projection display step that receives the object position, the view point position, and the filtered frame of the moving image as inputs, projects the object on a two-dimensional display projection plane based on the object position and the view point position, maps the frame of the moving image to a projection plane of the object, and displays the resulting image.
According to a fourteenth embodiment of the present invention, further to the method for controlling moving image reproduction quality of the thirteenth embodiment, the distance calculation step further receives coordinates on the display projection plane in the three-dimensional virtual space as an input, and finds an inclination angle of the object with respect to the display projection plane based on the object position, the view point position, and the coordinates on the display projection plane in the three-dimensional virtual space, and the filter selecting step updates the filter selected according to the distance into a filter corresponding to the distance and the inclination angle, on the basis of a preset table or formula showing a relationship between inclination angles and filters used for enlarging or reducing the moving image to be mapped to the object.
According to a fifteenth embodiment of the present invention, there is provided a method for controlling the reproduction quality of a moving image which is reproduced from a fusion of the moving image and three-dimensional computer graphics in a three-dimensional virtual space. The method of this embodiment comprises: an object generation step that receives three-dimensional object data relating to an object to which the moving image is to be mapped as an external input, and places the object in the three-dimensional virtual space to generate the object based on the three-dimensional object data; an object position determination step that receives data relating to the object placed in the three-dimensional virtual space as an input from the object generation step, moves the object according to an external user input, and thereby determines the object position in the three-dimensional virtual space; a view point position determination step that receives the determined object position as an input, moves a view point in the three-dimensional virtual space according to an external user input, and thereby determines a view point position according to correlation between the object position and the view point position; a perspective projection step that receives the object position and the view point position as inputs and projects the object on a two-dimensional display projection plane based on the object position and the view point position; an area calculation step that receives data relating to a projection plane of the perspectively projected object, and calculates an area of the projection plane; a filter selecting step that receives the calculated area as an input, and selects a filter corresponding to the input area on the basis of a preset table or formula showing a relationship between areas of the projection plane of the object and filters used for enlarging or reducing the moving image to be mapped to the projection plane; a filter control step that receives information about the selected filter and performs control so that a frame of the moving image to be mapped to the object is enlarged or reduced by using the selected filter; a moving image generation step that receives moving image data input externally, and reproduces the frame of the moving image from the moving image data; a filtering step that receives the frame of the moving image as an input from the moving image generation step and filters the frame in accordance with the control executed in the filter control step; and a display step that maps the frame of the moving image as an input from the moving image generation step to the two-dimensional display projection plane on which the object has been perspectively projected as an input from the perspective projection step, and displays the resulting image.
According to a sixteenth embodiment of the present invention, further to the method for controlling moving image reproduction quality of the thirteenth embodiment or the fifteenth embodiment, the area calculation step further receives the object position as an input from the perspective projection step, and the filter selecting step, when three-dimensional object data whose object size has been changed is given to the object generation step as an external input, further receives the object position as an input from the area calculation step, recognizes that the object size has been changed, from the object position, and updates the filter selected according to the area into a filter corresponding to the changed object size, on the basis of a preset table or formula showing a relationship between object sizes and filters used for enlarging or reducing the moving image to be mapped to the object.
As should be the forgoing, according to the first and ninth embodiments of the present invention, the object position and the view point position in the three-dimensional virtual space are obtained to calculate the distance from the view point position to the object, the frame rate of the moving image to be mapped to the object is determined according to the distance and on the basis of the preset table or formula, and the frame of the moving image with the frame rate reduced is generated from the moving image data externally input according to the determined frame rate and mapped to the object. Therefore, when a display portion of the object on the display projection plane is smaller, the frame rate of the moving image of the object is reduced. Thereby, without making the user feel degradation of quality of the reproduced moving image on the display screen, the amount of operation in decoding for generating the moving image or texture mapping can be reduced.
According to the second and tenth embodiments of the present invention, when the inclination angle of the object with respect to the display projection plane is changed, it is decided that the inclination has been changed, from the object position, the view point position, and the three-dimensional coordinates on the display projection plane, and the frame rate is determined according to the changed inclination and on the basis of the preset table or formula showing the relationship between the inclinations and the frame rates. Since the display portion on the display projection plane becomes smaller with an increase in the inclination angle of the object with respect to the display projection plane, the frame rate of the moving image of the object is adjusted by reducing it. Thereby, without making the user feel degradation of quality of the reproduced moving image on the display screen, the amount of operation in decoding for generating the moving image or texture mapping can be reduced.
According to the third and eleventh embodiments of the present invention, the area of the object on the display projection plane is calculated based on the two-dimensional image data obtained by performing perspective transformation of the object according to the object position and the view point position in the three-dimensional virtual space, and the frame rate of the moving image to be mapped to the object is determined according to the area and on the basis of the preset table or formula, and the frame of the moving image with the frame rate reduced is generated from the moving image data externally input according to the determined frame rate and mapped to the object. Therefore, when the display portion of the object on the display projection plane is smaller, the frame rate of the moving image of the object is reduced. Thereby, without making the user feel degradation of quality of the reproduced moving image on the display screen, the amount of operation in decoding for generating the moving image or texture mapping can be reduced.
According to the fourth and twelfth embodiments of the present invention, when the size of the object in the three-dimensional virtual space is changed, it is decided that the size of the object has been changed, from the object position (three-dimensional coordinates of the object), and the frame rate is determined on the basis of the table or formula reselected based on the size from the preset tables or formulas. Therefore, according to change of the size of the object in the three-dimensional virtual space, the frame rate of the moving image of the object is adjusted. Thereby, without making the user feel degradation of quality of the reproduced moving image on the display screen, the amount of operation in decoding for generating the moving image or texture mapping can be reduced.
According to the fifth and thirteenth embodiments of the present invention, the object position and the view point position in the three-dimensional virtual space are obtained to calculate the distance from the view point position to the object, the filter used for filtering that enlarges or reduces the moving image to be mapped to the object is selected according to the distance and on the basis of the preset table or formula, and the frame of the moving image generated from the moving image data externally input is processed by using the selected filter and mapped to the object. Therefore, when the display portion of the object on the display projection plane is smaller, the image quality of the moving image of the object is reduced. Thereby, without making the user feel degradation of quality of the reproduced moving image on the display screen, the amount of calculations required for filtering that enlarges or reduces the moving image can be reduced.
According to the sixth and fourteenth embodiments of the present invention, when the inclination of the object with respect to the display projection plane is changed, it is decided that the inclination has been changed, from the two-dimensional coordinates of the object on the display projection plane, and the filter according to the changed inclination is selected on the basis of the table showing the correspondence between inclinations and filters to-be-selected. Therefore, by controlling the filtering for enlarging or reducing the moving image of the object, the amount of calculations required for filtering that enlarges or reduces the moving image can be reduced, without making the user feel degradation of quality of the reproduced moving image on the display screen.
According to the seventh and fifteenth embodiments of the present invention, the object position and the view point position in the three-dimensional virtual space are obtained, the area of the object on the display projection plane is calculated based on the two-dimensional image data obtained by performing perspective transformation of the object, the filter used for filtering that enlarges or reduces the moving image to be mapped to the object is selected according to the area and on the basis of the preset table or formula, and the frame of the moving image generated from the moving image data externally input is processed by using the selected filter and mapped to the object. Therefore, when the display portion of the object on the display projection plane is smaller, the image quality of the moving image of the object is reduced. Thereby, without making the user feel degradation of quality of the reproduced moving image on the display screen, the amount of calculations required for filtering that enlarges or reduces the moving image can be reduced.
According to the eighth and sixteenth embodiments of the present invention, when the size of the object in the three-dimensional virtual space is changed, it is decided that the size has been changed, from the object position (three-dimensional coordinates), and the filter according to the changed size is selected on the basis of the table showing the correspondence between object sizes and filters to-be-selected. Therefore, by controlling the filtering for enlarging or reducing the moving image of the object according to change of the size of the object in the three-dimensional virtual space, the amount of calculations required for filtering that enlarges or reduces the moving image can be reduced, without making the user feel degradation of quality of the reproduced moving image on the display screen.